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PREFACE 


This  report  provides  the  results  of  a  study  to  assess  the  effectiveness 
of  using  micro  worlds  to  train  Army  logisticians.  This  study  was  con¬ 
ducted  as  part  of  a  larger  project  to  design  training  for  a  new  logistics 
organization,  the  Theater  Support  Command.  The  work  was  spon¬ 
sored  by  the  U.S.  Army  Combined  Arms  Support  Command.  It 
should  interest  both  those  who  are  involved  with  military  logistics 
and  those  involved  with  training  and  education. 

The  project  was  carried  out  in  the  Manpower  and  Training  Program 
of  RAND  Arroyo  Center,  a  federally  funded  research  and  develop¬ 
ment  center  sponsored  by  the  United  States  Army. 
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For  more  information  on  RAND  Arroyo  Center,  contact  the  Director 
of  Operations  (telephone  310-393-0411,  extension  6500;  FAX  310- 
451-6952;  e-mail  donnab@rand.org),  or  visit  the  Arroyo  Center's  Web 
site  at  http://www.rand.org/organization/ard/. 
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SUMMARY 


BACKGROUND 

The  Army  faces  new  challenges  in  training  its  logistics  managers.  As 
the  Army  evolves  into  a  force-projection  Army,  the  design  and  man¬ 
agement  of  large-scale  logistics  systems  assume  increasing  impor¬ 
tance.  However,  these  skills  are  difficult  to  train,  because  large 
deployments  occur  infrequently  and  thus  opportunities  to  design 
and  manage  systems  also  occur  infrequently.  Furthermore,  most  of 
the  people  who  work  in  high-level  logistics  management  organiza¬ 
tions  are  in  the  Army  Reserve,  and  they  train  and  practice  their  skills 
part  time.  Also,  reserve  units  tend  to  be  spread  across  large  geo¬ 
graphic  areas,  which  makes  it  difficult  to  regularly  train  complete 
units  at  one  time.  Finally,  the  reserves  experience  considerable 
turnover  in  personnel,  which  poses  a  formidable  challenge  to  main¬ 
taining  continuity  and  creates  a  constant  demand  to  train  new 
members.  The  Army’s  ongoing  revision  of  its  logistics  doctrine  exac¬ 
erbates  these  challenges. 

A  DIFFERENT  APPROACH 

The  Army  needs  a  way  to  come  to  grips  with  these  training  chal¬ 
lenges.  One  approach,  the  use  of  microworlds,  offers  a  potential 
solution.  A  microworld  is  a  computer-based  learning  tool  that 
enables  an  individual  to  interact  with  a  specific  environment  much 
in  the  way  that  a  flight  simulator  enables  a  pilot  to  practice  a  wide 
range  of  aerial  maneuvers.  Microworlds  can  teach  both  content  and 
process. 
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RAND  has  successfully  employed  microworlds  to  teach  complex 
academic  skills.  That  experience,  combined  with  the  work  of  others, 
led  RAND  researchers  to  hypothesize  that  microworlds  could  assist 
the  Army  with  its  problem  of  training  logistics  design  and  manage¬ 
ment.  Specifically,  it  appeared  feasible  for  microworlds  to  simulate  a 
dynamic  theater-level  management-distribution  system  that  could 
both  enable  exploratory  learning  (i.e.,  discovering  what  kinds  of 
problems  will  crop  up)  and  train  on-the-job  skills.  Such  a  capability 
would  offer  many  advantages.  It  could  be  used  to  train  new  mem¬ 
bers  without  waiting  for  infrequently  conducted  large-scale  exer¬ 
cises;  it  could  also  be  used  for  interunit  exercises  conducted  in  sev¬ 
eral  different  locations  simultaneously  using  the  Internet.  The 
microworlds  would  be  far  less  expensive  and  more  flexible  than 
large-scale  simulations  that  run  on  mainframe  computers. 

One  problem  is  that  we  have  little  empirical  data  on  the  effectiveness 
of  using  microworlds  to  teach  complex  management  skills.  Demon¬ 
strating  such  effectiveness  is  difficult,  but  before  the  Army  invests  in 
microworlds,  it  would  be  prudent  to  determine  whether  they  can 
provide  the  expected  benefits. 

THE  STUDY 

To  provide  such  a  demonstration,  RAND  developed  a  quasi- experi¬ 
mental  procedure  to  test  the  effectiveness  of  using  microworlds  to 
train  soldiers  in  the  design  and  management  of  logistics  systems. 
The  study  was  conducted  as  part  of  a  larger  RAND  Arroyo  Center  ef¬ 
fort  to  design  training  for  an  emerging  Army  organization,  the 
Theater  Support  Command,  and  one  of  its  subordinate  elements,  the 
Distribution  Management  Center. 

The  study  consisted  of  a  three-hour  curriculum  built  around  a 
microworld  model  that  simulates  a  simplified  version  of  a  distribu¬ 
tion  management  process.  Sixty-five  Army  reservists  from  theater- 
level  distribution  management  organizations  participated  in  training 
and  demonstration  sessions  and  then  completed  a  skills  assessment. 
The  training  and  assessment  approach  included  several  features  de¬ 
signed  to  test  the  effectiveness  of  the  microworld-based  curriculum 
strategy  while  minimizing  threats  to  validity.  However,  no  compari¬ 
son  was  made  between  the  microworld-based  training  and  other 
training  methods  on  either  a  cost  or  effectiveness  basis. 
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THE  RESULTS 

The  results  indicate  that  microworlds  can  effectively  train  complex 
management  skills.  We  used  two  categories  of  measures  to  deter¬ 
mine  the  effects  of  the  training:  objective  and  subjective. 

Objective  Measures 

On  average,  after  training,  participants  were  able  to  identify  more 
problematic  trends  in  data  than  they  were  before  training.  Im¬ 
provements  in  identifying  the  effects  of  the  trends  were  smaller  but 
still  significant.  We  examined  the  effects  of  a  number  of  variables  on 
the  test  results:  (1)  outcome  measures,  (2)  experimental  control  vari¬ 
ables,  and  (3)  demographic  variables.  None  correlated  even  moder¬ 
ately  with  the  change  from  pretest  to  posttest;  however,  a  few  corre¬ 
lated  with  pretest  performance,  posttest  performance,  or  both. 

Subjective  Measures 

The  subjective  measures  mirrored  the  objective  ones.  Participants 
believed  that  their  understanding  of  key  concepts  related  to  distri¬ 
bution  management  increased  during  the  course  of  the  three-hour 
training  session.  Specifically,  they  reported  increases  in  their 
understanding  of  how  to  measure  a  theater  distribution  system,  how 
to  evaluate  alternative  distribution  policies,  the  effects  of  policies 
over  time  and  on  other  organizations,  and  how  to  manage  a  general 
supply  distribution  network. 

OBSERVATIONS 

Most  worthy  of  note  in  this  study  is  the  finding  that  the  use  of 
microworld  training  led  to  significant  improvement  in  identifying 
trends  in  data  and  in  analyzing  the  effects  of  those  trends.  This  is 
particularly  significant  because  the  test  environment  was  challeng¬ 
ing,  and  the  learning  of  a  substantial  amount  of  material  took  place 
in  a  short  time.  Of  the  three  hours,  only  about  one  hour  was  devoted 
to  time  on  the  task.  The  rest  was  devoted  to  introductory  material, 
assessments,  and  an  explanation  of  the  mechanics  of  the  microworld 
model. 
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Should  microworld-based  curricula  continue  to  demonstrate 
promise,  a  number  of  directions  should  be  pursued  to  support  their 
use.  First,  a  reusable  library  of  scenarios  that  can  be  assessed  via 
networks  should  be  developed.  Also,  having  network-based  access  to 
microworld  curricula  (e.g.,  via  the  World  Wide  Web)  would  enable 
collaborative  training.  Microworld-based  training  seems  to  offer 
considerable  promise  for  training  the  sorts  of  complex  management 
skills  needed  by  logistics  system  designers  and  managers. 
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Chapter  One 

INTRODUCTION 


BACKGROUND 

As  the  U.S.  Army  evolves  into  a  “force-projection  Army/'  the  re¬ 
quirement  to  deploy  quickly  and  conduct  missions  away  from  its 
garrison  locations  places  increasing  importance  on  effective  logistics 
design  and  management  skills.  Such  skills  not  only  will  have  to  be 
executed  effectively  on  short  notice,  they  also  will  have  to  be  applied 
in  increasingly  information -intensive  and  geographically  distributed 
environments,  similar  in  many  ways  to  the  environments  faced  by 
commercial  logistics  organizations.  But  military  organizations  face 
special  challenges  in  training  logistics  managers  that  do  not  appear 
in  most  corporate  training  arenas.  Military  logisticians  must  be  pre¬ 
pared  to  design,  build,  and  operate  distribution  systems  that  provide 
comprehensive  support  to  hundreds  of  thousands  of  deployed  peo¬ 
ple  and  their  equipment.  There  are  a  number  of  challenges: 

•  These  distribution  systems  do  not  exist  most  of  the  time  because 
large-scale  deployments  are  infrequent,  so  opportunities  for  ef¬ 
fective  on-the-job  training  are  limited. 

•  Personnel  who  work  in  U.S.  Army  logistics  management  organi¬ 
zations  are  largely  Army  Reserve  soldiers  who  get  very  little 
training  time  focused  on  their  logistics  management  skills 
(Bondanella  et  al.,  1997). 

•  There  is  a  high  personnel  turnover  or  “churn"  in  Army  Reserve 
organizations  (Bondanella  et  al.,  1997),  so  training  newcomers  is 
an  ongoing  task. 
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•  The  home  locations  of  reservists  in  these  organizations  can  be 
geographically  distributed  over  hundreds  of  miles.  Hence,  train¬ 
ing  regularly  with  teammates  or  sister  organizations  is  difficult. 

These  challenges  are  made  more  difficult  by  the  ongoing  evolution  of 
Army  doctrine  covering  how  these  logistics  organizations  are  de¬ 
signed  and  staffed,  the  technologies  they  will  use,  and  their  business 
practices.  Two  important  goals  of  these  new  practices  are  (1)  more 
effective  distribution  processes  and  (2)  efficiently  measured  end-to- 
end — or  systemic — performance.  This  “process  perspective”  reflects 
a  major  change  in  orientation  for  an  organization  that  has  tradition¬ 
ally  stressed  a  more  functionally  driven,  project-oriented  approach, 
where  performance  is  measured  based  on  how  well  specific  sub- 
segments  of  the  process  perform.  The  Army  is  trying  to  follow  the 
lead  of  prominent  private-sector  logistics  organizations  by  shifting 
its  focus  to  activity  or  process  measures  and  systemic  performance. 


A  DIFFERENT  APPROACH  TO  LOGISTICS  MANAGEMENT 
TRAINING 

Using  Micro  worlds  for  Training 

One  training  tool  that  has  gained  prominence  with  increasing  em¬ 
phasis  on  systemic  performance  is  the  microworld  model.  Lawler 
(1987)  describes  a  microworld  as  a  virtual,  interactive  subset  of  the 
world,  limited  to  a  specific  domain.  Microworlds  offer  interesting 
environments  that  learners  can  explore  interactively.  The  purpose  of 
such  environments  is  to  teach  understanding  of  content,  such  as 
how  gravity  works  (Smith,  1986),  as  well  as  to  teach  “habits  of  explo¬ 
ration  from  their  personal  lives  to  the  formal  domain  of  scientific 
theory  construction”  (Papert,  1980).  The  theoretical  educational 
strength  underlying  the  use  of  microworlds  is  that  exploration-based 
learning  is  highly  efficacious.  According  to  this  view,  an  individual 
who  sets  learning  goals  and  then  learns  by  exploring  and 
experimenting,  or  “learning  by  doing,”  richly  elaborates  and 
integrates  new  knowledge  into  current  knowledge. 

The  term  “microworld”  appears  to  have  originated  at  the  MIT  Artifi¬ 
cial  Intelligence  Laboratory  in  the  early  1970s  (Lawler,  1987)  and  was 
later  popularized  by  Seymour  Papert  (1980).  Although  a  microworld 
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can  be  built  using  any  enabler,  Papert  used  the  term  in  the  context  of 
a  “computer-based  learning  environment  for  children,  in  which  they 
could  program  the  environment,  see  how  it  responded,  and  draw  out 
their  own  understanding  of  the  principles  of  mathematical  relation¬ 
ships.”1  Accordingly,  Papert  developed  the  simple  but  powerful  Logo 
computer  programming  language,  which  emphasizes  graphics  and 
was  used  during  the  early  stages  of  developing  computer-based 
microworlds.2  Another  software  application — STELLA — has  been 
used  in  educational  contexts  since  the  early  1980s  to  support  ex¬ 
ploratory  learning  and  teach  systems  thinking.  STELLA  is  a  com¬ 
mercial  programming  language  oriented  on  educational  issues  in  the 
natural  and  social  sciences  and  the  humanities;  its  structure  parallels 
ithink,  a  software  product  oriented  on  business  issues.3  A  STELLA 
model  begins  with  the  computer  visualization  and  building  of  a 
stock-and-flow  diagram,  which  is  then  supported  by  underlying 
mathematical  algorithms.4  Work  by  Mandinach  and  her  colleagues 
(Mandinach,  1989;  Mandinach  and  Cline,  1994)  has  demonstrated 
how  to  integrate  this  category  of  modeling  and  simulations  into 
classroom  learning. 

Others  have  built  and  experimented  with  microworlds  (Shute  and 
Glaser,  1990;  Shute,  Glaser,  and  Raghavan,  1988;  Smith,  1986)  and 
generally  demonstrated  their  qualitative  and,  in  some  cases,  quanti¬ 
tative  efficacy  for  teaching  complex  skills.  Certainly,  inquiry-based 
learning  and  the  learning  tools  to  support  it  are  not  new  topics. 
Cuban  (1986)  and  Cohen  (1988),  among  others,  have  chronicled 
several  cycles  of  interest  in  inquiry  learning  among  academics  and 
educators.  Others,  like  Starr  (1994),  eloquently  point  out  the  poten- 


1  Peter  M.  Senge,  Art  Kleiner,  Charlotte  Roberts,  Richard  B.  Ross,  and  Bryan  J.  Smith, 
The  Fifth  Discipline  Fieldbook:  Strategies  and  Tools  for  Building  a  Learning  Organiza¬ 
tion,  New  York:  Doubleday,  1994,  p.  531. 

2 According  to  an  entry  for  Logo  at  www.techweb.com/encyclopedia/,  “Stemming  from  a 
National  Science  Foundation  project,  Logo  was  created  by  Seymour  Papert  in  the  mid 
1960s  along  with  colleagues  at  MIT  and  members  of  Bolt  Beranek  &  Newman. 
Originally  developed  on  large  computers,  it  has  been  adapted  to  most  personal  com¬ 
puters.” 

3These  software  modeling  programs  were  developed  by  High  Performance  Systems, 
Inc.,  and  are  described  on  their  Web  site,  www.hps-inc.com/customer_center/kb/ 
webhelp/hpsn.  htm. 

4 According  to  www.ncsa.uiuc.edu/edu/RSE/RSBindigo/stella.html,  “STELLA  II®  is  the 
acronym  for  Systems  Thinking  in  an  Experimental  Learning  Lab  with  Animation." 
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tial  mislearning  and  pitfalls  of  using  microworlds  to  teach  complex 
knowledge,  especially  with  models  that  cannot  be  inspected  by  the 
learner. 

Earlier  work  at  RAND  applied  microworlds  to  teach  complex  aca¬ 
demic  topics  (McArthur  and  Lewis,  1991;  McArthur,  Lewis,  and 
Bishay;  1995,  and  McArthur  and  Lewis,  1998).  The  work  of  Senge  et 
al.  (1994)  in  applying  microworlds  as  "management  flight  simula¬ 
tors”  was  a  logical  extension  of  discovery  learning  supported  by 
computer  simulations  to  teach  business  skills  and  understanding. 
Beyond  that,  it  appeared  to  us  that  connecting  microworlds  in  a  net¬ 
work  and  allowing  multiple  learners  to  collaborate  at  a  distance 
could  address  some  of  the  training  challenges  outlined  above.  Based 
on  the  background  just  described,  we  hypothesized  that  using 
microworlds  to  teach  Army  logistics  management  skills  could  poten¬ 
tially  provide  the  following: 

•  A  dynamic  simulated  theater-level  distribution  system  with 
which  to  engage  in  exploratory  learning  and  to  replicate  some  of 
the  benefits  of  on-the-job  training. 

•  Focused  individual  training  directly  relevant  to  the  skills  that 
Army  Reserve  soldiers  will  need  if  they  deploy  to  support  a  con¬ 
tingency. 

•  Readily  accessible  training  to  the  many  new  members  of  the  unit 
without  waiting  for  an  annual  training  exercise. 

•  Small-group  and  interorganizational  training  at  a  distance: 
Teams  can  interact  with  shared,  Web-based  microworlds  from 
their  home  sites  or  interact  with  other  organizations  using 
shared  microworlds. 

•  Simulations  that  are  less  expensive  to  develop  and  use  for  train¬ 
ing  than  the  current  large,  mainframe-based  systems  that  require 
support  staffs. 

Few  Empirical  Data  Are  Available 

Although  the  potential  benefits  of  using  microworlds  to  train  Army 
logistics  managers  sound  appealing,  there  is  little  empirical  data  on 
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the  effectiveness  of  microworlds  to  teach  complex  management 
skills  and  understanding. 

Empirically  demonstrating  the  educational  effectiveness  of 
microworlds  in  academic  settings  is  difficult.  Empiricists  argue  that 
understanding  and  skills  are  acquired  from  exploration  with 
microworlds  but  that  such  learning  is  difficult  to  test  explicitly,  and  it 
is  similarly  difficult  to  tease  out  the  effect  of  the  microworld  from 
other  aspects  of  the  new  training  materials.  Microworlds  are  not 
meant  to  be  used  in  a  vacuum,  so  other  pieces  of  the  curriculum  to 
support  the  guided  microworld  explorations  are  also  novel  to  the 
experimental  condition. 

Trying  to  teach  management  skills  to  adults  using  micro  worlds 
appears  to  be  empirically  documented  even  less  well.  Although  the 
importance  of  microworlds  for  teaching  critical  thinking  skills  to 
youth  and  adult  learners  is  argued  by  consulting  companies  and 
providers  of  simulation  tools  (e.g.,  Richmond,  1993),  there  appears  to 
be  little  hard  data  to  support  their  efficacy  or  their  current  level  of 
use  in  firms.  Nonetheless,  an  article  by  Gross  (1996)  included  a  pro¬ 
jection  done  by  the  Gartner  Group,  Inc.,  based  on  a  survey  of  organi¬ 
zations  using  business  process  reengineering  (BPR)  tools.  The  article 
reported  that  one  in  four  of  such  companies  are  using  or  planning  to 
use  products  with  advanced  simulation  or  animation  capabilities 
and  that,  according  to  the  Gartner  Group  report,  by  1998  over  half  of 
all  BPR  projects  would  use  some  form  of  simulation  or  animation. 

More  recently,  Salopek  (1998)  reported  that  60  percent  of  corpora¬ 
tions  surveyed  used  some  kind  of  simulation-based  training,  but 
again,  effectiveness  data  were  not  available.  Searches  of  the  Internet 
provide  a  number  of  simulation  and  consulting  service  provider  sites 
that  list  examples  of  the  simulations  they  have  built  for  clients. 
Glowing  accounts  of  client  satisfaction,  but  no  published  outcome 
studies,  can  be  found. 

FOCUS  OF  THE  STUDY 

In  sum,  there  is  growing  interest  in  the  use  of  micro  worlds  for  train¬ 
ing  complex  management  skills  but  few  reliable  studies  assessing 
their  effects.  The  goal  of  our  study  was  to  provide  some  quasi-exper- 
imental  empirical  evidence  that  would  aid  Army  logisticians  in  eval- 
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uating  the  usefulness  of  microworlds  to  teach  complex  logistics 
management  skills,  while  perhaps  informing  the  debate  over  the 
effectiveness  of  microworlds  as  training  tools.  However,  we  made  no 
comparison  between  microworld-based  training  and  other  training 
methods  on  either  a  cost  or  effectiveness  basis.  This  is  largely  due  to 
the  fact  that  there  are  no  other  current  curricula  to  teach  the  content 
for  comparison.  These  are  new  concepts  to  the  Army  logistics  com¬ 
munity,  so  there  is  no  natural  set  of  materials  and  methods  to  use  for 
an  alternative  training  condition. 

Our  study  was  conducted  as  part  of  an  effort  to  design  new  training 
for  an  emerging  organization  in  the  Army — the  Theater  Support 
Command  (TSC)— of  which  a  subordinate  organization,  the  Distri¬ 
bution  Management  Center  (DMC),  is  responsible  for  managing  the 
flow  of  people  and  materiel  throughout  the  theater  of  operations. 
Although  learners  would  ideally  spend  weeks  manipulating  a 
microworld  model  to  test  hypotheses  and  gain  system  understand¬ 
ing,  we  were  severely  constrained  in  terms  of  the  time  available.  We 
had  only  a  three-hour  window  in  which  to  conduct  the  evaluation 
study.  We  therefore  designed  a  three-hour  training  curriculum 
around  a  microworld  model  that  represents  a  simplified  version  of 
the  distribution  management  process.  During  the  training  session, 
the  model  was  run  on  a  large  projection  screen.  Participants  were 
given  many  opportunities  to  discuss  hypotheses  about  the  system, 
but  due  to  the  time  constraint,  it  was  not  feasible  for  them  to  ma¬ 
nipulate  the  model  on  an  individual  basis.  The  main  goal  of  the  ses¬ 
sion  was  to  teach  the  learners  about  the  consequences  of  their 
decisions  over  time  and  across  the  entire  system.  In  reaching  this 
goal,  it  was  necessary  to  also  provide  soldiers  with  some  background 
knowledge  and  basic  skills  relevant  to  performing  as  part  of  a  TSC. 

ORGANIZATION  OF  THE  DOCUMENT 

The  remainder  of  this  report  describes  our  microworld-based  train¬ 
ing  evaluation  study  with  U.S.  Army  Reserve  soldiers  learning  logis¬ 
tics  management  skills.  We  then  present  the  results  of  that  study  and 
close  with  a  discussion  of  how  we  believe  our  models  and  curriculum 
can  be  broadened  and  improved  to  provide  better  learning  support 
for  locally  gathered  and  distributed  teams. 


Chapter  Two 

METHOD 


PARTICIPANTS 

Sixty- five  Army  reservists  from  theater- level  distribution  manage¬ 
ment  organizations  participated  as  part  of  a  two-week  annual  train¬ 
ing  event.  Table  1  is  a  display  of  some  of  the  important  demographic 
characteristics  of  the  participants. 

MATERIALS  AND  APPARATUS 

All  participants  were  given  a  “Battle  Book”  at  the  start  of  training;  it 
contained  a  sheet  explaining  participants’  rights  as  human  subjects 
in  an  experiment,  assessment  forms,  and  paper  copies  of  the  training 
slides. 

A  dynamic  microworld  model  was  also  used  during  training.  The 
microworld  model  operates  as  a  computer  simulation  superimposed 
on  a  map  that  represents  the  distribution  of  supplies  on  the  Japanese 
island  of  Hokkaido.  During  training,  the  simulation  allows  the  user 
to  monitor  the  performance  of  the  distribution  network  over  a  period 
of  24  days  by  way  of  animation  and  numerical  displays.  This  occurs 
over  92  time  steps  in  the  model  so  that  a  single  time  step  represents  a 
six-hour  period.  The  model  allows  users  to  control  the  distribution 
policy  throughout  the  network  and  to  evaluate  its  long-term  effects. 
Variables  that  the  user  can  modify  are  collected  in  a  notebook  that  is 
accessible  from  a  pull-down  display. 

Figure  1  shows  the  model  as  it  appears  to  the  user  for  supply  distri¬ 
bution  on  Hokkaido.  On  the  map,  each  area  of  interest  is  a  node, 
bearing  the  name  of  the  principal  city  or  camp  (e.g.,  Nishiokoppe). 
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Table  1 

Demographics  of  the  Participants 


Demographic  Variable 

Number  of  Participants 

Enlisted /officer 

Total  enlisted  (E5-E9) 

17  (90%  E7-E9) 

Total  officer  (02-06) 

47  (80%  04-05) 

Civilian 

1 

Time  in  permanent  unit 

One  year  or  less 

20 

Between  1  and  3  years 

17 

Over  3  years 

28 

Time  in  current  job  position 

One  year  or  less 

35 

Between  1  and  2  years 

14 

Over  2  years 

16 

Time  in  active  duty  logistics  unit 

Never 

36 

Three  years  or  less 

12 

Between  3  and  10  years 

9 

Over  10  years 

8 

Degree  of  civilian  education 

High  school  diploma 

5 

Associate  degree 

8 

Baccalaureate  degree 

25 

Master’s  degree 

17 

Other  graduate /professional  degree 

10 

Each  node  also  has  a  number  used  to  designate  it  in  the  notebook. 
Within  each  node  is  a  vertical  bar  that  serves  as  a  graphical  display  of 
the  supply  level  at  the  node.  Next  to  the  bar  is  a  number  that  pro¬ 
vides  the  same  information.  If  the  level  of  supplies  exceeds  set  limits, 
a  red  square  above  the  bar  lights  up. 

Supplies  originate  in  CONUS  and  in  Japan  and  travel  by  ship  to  the 
sea  port  of  debarkation  (SPOD)  at  Atsukeshi,  the  SPOD  at  Kushiro,  or 
the  aerial  port  of  debarkation  (APOD)  at  Kushiro.  Supplies  then 
travel  either  over  land  or  by  air.  Over  land,  they  are  taken  to  Teshi- 
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Figure  1 — Microworld  Model 


kaga.  By  air,  they  are  taken  directly  to  Niufu  from  the  APOD  at 
Kushiro.  At  Teshikaga,  some  supplies  are  routed  to  Kamiyubetsu  and 
others  to  Nishiokoppe.  Kamiyubetsu  and  Nishiokoppe  feed  their 
supplies  to  Omu  and  Niufu,  respectively. 

Supplies  are  carried  between  nodes  by  transportation  resources 
rather  than  by  individual  vehicles.  A  transportation  resource  is  an 
aggregate  of  vehicles  and  may  correspond  to  a  train,  a  convoy  of 
trucks,  or  a  number  of  transport  aircraft  or  ships.  When  the  model 
operates,  a  transportation  resource  appears  as  a  colored  circle  that 
moves  along  the  transportation  corridors  that  connect  nodes.  If  the 
transportation  resource  is  loaded,  the  pattern  in  the  circle  is  solid.  If 
the  resource  is  empty,  the  pattern  in  the  circle  is  hatched.  Each 
transportation  resource  is  uniquely  labeled  by  color  so  that  the  user 
may  keep  track  of  its  position  and  status. 
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DESIGN  AND  PROCEDURE 

Two  identical  demonstration  training  sessions  were  conducted,  one 
in  the  morning  and  one  in  the  afternoon.  Thirty-one  people  partici¬ 
pated  in  the  morning  session,  and  34  participated  in  the  afternoon. 
A  single  session,  including  both  training  and  assessment,  was  de¬ 
signed  to  be  completed  in  three  hours.  Shortly  before  the  start  of  the 
session,  all  participants  were  asked  to  provide  demographic  infor¬ 
mation  about  themselves,  including  their  ranks,  levels  of  civilian  and 
military  education,  and  participation  in  related  pilot  studies.  After 
completing  the  demographics  form,  they  were  asked  to  close  their 
Battle  Books  and  wait  for  the  training  session  to  begin. 

Participants  in  the  training  session  were  randomly  assigned  to  one  of 
two  groups:  Full  Assessment  and  Posttest  Control.  For  participants 
in  the  Full  Assessment  group,  the  session  began  with  the  administra¬ 
tion  of  a  pretest  to  measure  the  initial  level  of  participants’  skills.  The 
pretest  was  followed  by  training  and  then  by  a  posttest.  The  Posttest 
Control  group  received  identical  training  but  completed  only  the 
posttest.  The  purpose  of  the  control  condition  was  to  determine 
whether  experience  with  the  pretest  would  itself  affect  performance 
on  the  posttest.  Few  participants  were  needed  to  test  such  an  effect; 
thus,  there  were  52  participants  in  the  Full  Assessment  group  and 
only  eight  in  the  Posttest  Control  group.  The  remaining  five  partici¬ 
pants  took  the  pretest  only  and  were  excluded  from  the  analysis. 
Following  the  description  below  of  the  training  session,  we  discuss 
the  details  of  the  assessment  strategy. 

Training 

The  training  session  focused  on  distribution  management  skills.  We 
identified  and  trained  participants  on  three  broad  categories  of  skills 
that  are  considered  by  subject  matter  experts  to  be  central  to  per¬ 
formance  as  part  of  a  TSC:  (a)  basic  system  knowledge  and  skills,  (b) 
measurement  and  trend  identification  skills,  and  (c)  course-of-action 
analysis  skills.  We  used  a  mix  of  methods  to  teach  these  skill  sets, 
and  the  complexity  of  the  methods  was  commensurate  with  the 
complexity  of  the  skill. 

In  teaching  basic  system  knowledge,  we  relied  almost  exclusively  on 
lectures.  Here,  we  presented  a  short  lecture  (10  minutes)  on  the 
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Figure  2 — Force  Dispositions  on  Hokkaido 


missions  and  organization  of  the  TSC  and  DMC.  Discussion  during 
this  phase  was  minimal. 

The  second  section  of  the  training  session  was  a  15-minute  group 
discussion  of  metrics  and  measurement.  At  the  start  of  this  section, 
we  presented  a  hypothetical  scenario  and  distribution  network  on 
the  island  of  Hokkaido  (depicted  in  Figure  2).  The  scenario  was  a 
hypothetical  joint  bilateral  operation  between  the  Americans  and  the 
Japanese,  with  the  American  forces  on  the  east  side  of  the  island, 
facing  north,  and  the  Japanese  Defense  Forces  on  the  west  side.  The 
objective  was  to  push  the  “Orange  Forces”  (the  enemy)  out  of  the 
northern  part  of  the  island,  and  the  logistician's  job  was  to  manage 
the  flow  of  a  general  class  of  supply  from  the  ports  to  the  Division 
Support  Commands  (DISCOMs) — the  retail  organizations  that  sup¬ 
ply  the  tactical  units.  The  group  discussion  of  measurement  was 
structured  around  several  issues,  including  identification  of  the  cus¬ 
tomer^),  selecting  appropriate  metrics,  and  thinking  about  different 
time  horizons. 

The  final  and  longest  training  segment  focused  on  the  analysis  of  a 
distribution  network.  In  this  segment,  as  we  began  to  train  the  more 
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complex  skill  of  managing  a  dynamic  system,  we  emphasized  the  use 
of  microworld  simulations.  The  simulations  were  deliberately  sim¬ 
plified  to  teach  a  few,  very  specific  concepts  and  skills,  including  how 
to  recognize  and  characterize  patterns  in  data  and  how  to  approach 
the  analysis  of  different  courses  of  action. 

At  the  start  of  this  final  section,  we  reintroduced  the  scenario  and 
operations  area  (Figure  2).  We  then  presented  a  schematic  of  the 
general  supply  distribution  network  on  the  same  map  (Figure  3). 
Next  we  introduced  our  highly  simplified  microworld  model  of  sup¬ 
ply  distribution  and  explained  to  participants  the  mechanics,  graph¬ 
ics,  and  animation  of  the  model. 

After  a  15-minute  break,  the  first  dynamic  network  was  presented.  It 
was  a  hub-and-spoke  system  with  the  hub  at  a  tank  farm  between 
points  of  debarkation  and  DISCOMs  (Figure  4).  After  describing  the 
network  in  detail,  we  asked  participants  as  a  group  to  generate  pre¬ 
dictions  as  to  the  performance  of  the  system  and  to  explain  the 
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Figure  3 — Distribution  Network 
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rationale  for  their  responses.  Following  the  prediction  exercise,  we 
ran  a  microworld  simulation  of  the  hub-and-spoke  system  that 
enabled  participants  to  see  the  effects  of  a  pure  hub-and-spoke  pol¬ 
icy  over  time.  The  outcomes  they  predicted  and  subsequently  wit¬ 
nessed  included  a  bottleneck  at  the  hub  and  unacceptably  low  sup¬ 
ply  levels  at  the  DISCOMs  (“starving  customers”). 

Before  running  our  simulation  of  a  pure  direct- delivery  system  from 
the  ports  to  the  DISCOMs,  we  showed  participants  a  representative 
diagram  (depicted  in  Figure  5)  and  asked  not  only  for  their  predic¬ 
tions,  but  also  for  comparisons  to  the  performance  of  the  hub-and- 
spoke  system.  We  then  moved  to  a  discussion  about  important  fac¬ 
tors  to  consider  in  evaluating  a  course  of  action  (COA),  e.g.,  hub-and- 
spoke  or  direct  delivery. 

Following  the  prediction  exercise,  participants  watched  a  simulation 
run  of  the  direct- delivery  system  and  noted  both  positive  and  nega¬ 
tive  aspects  of  a  switch  to  direct  delivery.  The  DISCOMs  received 


Figure  4 — Microworld  Model  of  Hub-and-Spoke  Scenario 
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Figure  5 — Microworld  Model  of  Direct-Delivery  Scenario 


supplies  in  a  more  timely  manner,  but  intermediate  nodes  were 
starved,  and  long  transit  times  led  to  backups  at  the  ports.  Following 
this  final  run,  we  displayed  graphs  of  the  supply  levels  of  several 
nodes  over  time,  reviewed  factors  to  consider  in  evaluating  COAs, 
administered  a  25-minute  posttest,  and  then  presented  a  more 
complex  model  with  a  built-in  feedback  loop.  The  final  feedback 
model  was  included  only  as  a  demonstration  of  model  capabilities 
and  was  not  a  target  of  the  assessments.  Feedback  from  participants 
was  collected  in  the  final  five  minutes  of  the  session. 

Assessments 

For  the  Full  Assessment  group,  two  sets  of  assessments — a  pretest 
and  a  posttest — were  embedded  in  the  three-hour  session.  The  tests 
were  administered  immediately  before  and  after  training,  and  partic¬ 
ipants  were  given  30  minutes  to  complete  each  set. 


Method  15 


Both  subjective  and  objective  measures  of  performance  were  col¬ 
lected  during  each  test.  The  subjective  measures  consisted  of  partic¬ 
ipant  ratings  of  their  own  understanding  of  concepts  spanning  all 
three  categories  of  skills:  basic  system  knowledge,  trend  identifica¬ 
tion,  and  COA  analysis.  Less  than  five  minutes  was  required  to 
complete  the  subjective  questions. 

We  collected  objective  measures  of  skill  for  only  the  two  most  com¬ 
plex  categories:  trend  identification  and  COA  analysis.  Basic  system 
knowledge  was  not  tested,  because  it  consisted  mainly  of  informa¬ 
tion  that  could  be  referenced  and  did  not  require  problem-solving 
skill.  Participants  were  given  ten  minutes  to  answer  questions  per¬ 
taining  to  each  of  the  two  skill  categories — 20  minutes  for  the  two 
combined.  Free-response  feedback  was  also  collected  and,  along 
with  administrative  time,  accounted  for  the  remainder  of  the  30- 
minute  time  blocks. 

For  the  objective  test  questions,  participants  were  given  a  specific 
scenario  (e.g.,  Figure  6),  a  map  of  the  system  to  analyze  (Figure  7), 
and  a  table  of  data  indicating  levels  of  POL  (petroleum,  oil,  and  lubri¬ 
cants)  or  numbers  of  personnel  at  each  location  in  the  network  over 
a  period  of  ten  days  (Figure  8).  We  asked  participants  to  answer  two 
main  questions  based  on  the  information  provided.  Question  1 
asked  participants  to  identify  problematic  patterns  in  the  data,  and 
Question  2  involved  COA  analysis  (see  Figure  9). 

The  initial  test  scenario  described  a  poorly  functioning  hub-and- 
spoke  system,  although  it  was  left  to  participants  to  recognize  that  it 
was  not  performing  well.  In  devising  the  test  materials,  we  built  sev¬ 
eral  problematic  trends  or  patterns  into  the  data,  some  more  subtle 
than  others,  and  asked  participants  to  identify  them  (Question  1). 
Before  presenting  the  question  on  COA  analysis,  we  added  a  short 
continuation  to  the  scenario  indicating  that  the  commander  was 
unhappy  with  the  hub -and -spoke  system  and  was  considering  a 
switch  to  direct  delivery.  Participants  were  asked  to  list  the  factors 
they  would  consider  in  deciding  whether  to  switch  to  a  direct- 
delivery  system  (Question  2). 


16  Exploring  Microworld  Models  to  Train  Army  Logistics  Management  Skills 


The  2nd  COSCOM  commander  is  using  an  intermediate 
petroleum  tank  farm  located  at  Bihiro,  containing  approxi¬ 
mately  3.2  million  gallons  of  JP8  fuel,  to  support  both  the 
combat  and  combat  support  elements  within  the  boundaries 
of  2nd  Corps.  Army-owned  line-haul  fuel  tankers  provide 
resupply  to  the  farm  from  head  terminals  located  at  Shibetsu 
and  Shiranuka.  Once  the  fuel  arrives  at  the  tank  farm, 
petroleum  handlers  download  it  into  bags  and  later  issue  it  to 
5,000-gallon  fuel  tankers  for  delivery  to  corps-level  fuel 
system  supply  points  (FSSPs).  Once  there,  the  fuel  is  stored 
in  the  FSSPs  until  it  is  delivered  to  divisional  forces  by 
divisional  fuel  tankers. 


Figure  6 — POL  Scenario  Used  in  Testing 


Figure  7 — Map  of  POL  System 
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Supply  levels  (in  thousands  of  gallons) 


Targets: 

6,000 

6,000 

3,200 

60 

60 

30 

30 

OPDS 

OPDS 

Tank  Farm 

FSSP 

FSSP 

DISCOM 

DISCOM 

Shiranuka 

Shibetsu 

Bihiro 

Nayoro 

Okoppe 

Kunneshiri 

Esashi 

Day  1 

1,000 

1,000 

0 

0 

0 

0 

0 

Day  2 

1,500 

1,500 

1,000 

30 

30 

0 

20 

Day  3 

2,000 

2,500 

1,500 

15 

20 

5 

10 

Day  4 

3,500 

3,000 

2,000 

20 

30 

10 

5 

Day  5 

3,000 

4,000 

1,500 

10 

20 

5 

25 

Day  6 

5,000 

3,500 

3,000 

25 

15 

10 

15 

Day  7 

6,500 

5,000 

3,500 

35 

10 

5 

10 

Day  8 

6,000 

6,000 

4,000 

45 

30 

10 

7 

Day  9 

6,800 

5,000 

4,500 

60 

20 

10 

20 

Day  10 

4,000 

4,000 

4,000 

60 

15 

5 

15 

Figure  8 — Data  for  POL  Scenario 


Question  1:  Trend  Identification 

What  trends  or  problematic  patterns  do  you  detect  currently 
or  see  developing  in  this  POL  network,  and  where  specifically 
do  they  occur? 

Question  2:  COA  Analysis 

The  COSCOM  commander  is  concerned  about  the  low  fuel 
supply  to  divisional  forces  at  Kunneshiri.  He  is  therefore 
considering  switching  to  a  system  of  direct  delivery  from  the 
head  terminal  at  Shiranuka  to  the  DISCOM  at  Kunneshiri. 

He  asks  for  your  input.  List  the  factors  you  would  consider 
in  deciding  whether  to  switch  to  the  new  system. 


Figure  9 — Sample  Assessment  Questions 
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We  also  included  a  simple  question  testing  a  participant’s  profi¬ 
ciency  in  reading  data  from  tables  of  numbers  (Question  3}  to  ensure 
that  performance  on  the  questions  of  interest  did  not  reflect  the 
absence  of  this  prerequisite  skill.  Data  from  participants  who  failed 
to  answer  the  question  correctly  were  analyzed  separately. 

Each  participant  was  randomly  assigned  to  a  single  test  vignette, 
either  POL  or  personnel.  The  POL  distribution  vignette  focused  on 
the  distribution  of  fuel  between  different  locations  in  the  network. 
The  personnel  replacement  vignette  described  the  flow  of  personnel 
between  locations.  Half  of  the  participants  were  exposed  to  the  POL 
vignette  and  half  to  the  personnel  vignette.  Each  vignette  had  two 
forms,  one  for  the  pretest  and  one  for  the  posttest.  Each  participant 
in  the  Full  Assessment  group  completed  both  forms  of  a  single 
vignette.  All  vignettes  were  located  on  the  east  side  of  the  island  of 
Hokkaido,  thereby  keeping  the  geography  constant  for  the  pretest, 
training,  and  posttest.  However,  different  city  names  were  used  for 
each  of  the  two  test  forms  and  for  the  curriculum  as  follows:  Test 
Form  A  for  POL  and  Test  Form  A  for  personnel  made  use  of  identical 
maps,  and  the  numbers  in  the  two  data  tables  differed  only  by  a  con¬ 
stant.  The  same  was  true  of  Test  Form  B  for  POL  and  Test  Form  B  for 
personnel.  Further,  Test  Forms  A  and  B  for  both  vignettes  featured 
identical  underlying  network  structures.  This  way,  the  vignettes  dif¬ 
fered  only  in  terms  of  cities  used,  the  items  being  moved  (POL  or 
people),  and  the  corresponding  magnitude  of  the  numbers.  They  did 
not  differ  in  context  or  structure.  The  order  of  the  test  forms  was 
counterbalanced  within  vignettes,  and  the  maps  for  all  the  test  forms 
differed  (in  terms  of  city  names  and  locations)  from  the  map  used 
during  training. 

Like  participants  in  the  Full  Assessment  group,  half  of  the  partici¬ 
pants  in  the  Posttest  Control  group  were  tested  with  the  POL 
vignettes,  and  the  other  half  with  the  personnel  vignettes.  Posttest 
forms  were  also  counterbalanced  across  groups  of  participants  in  the 
Posttest  Control  group.  During  administration  of  the  pretest  to  the 
Full  Assessment  group,  the  Posttest  Control  group  was  escorted  to  a 
separate  room  for  casual  discussion,  unrelated  to  the  training  topics. 


Method  19 


Scoring 

A  number  of  steps  were  required  for  the  scoring  and  analysis  of  the 
responses  to  the  objective  questions.  The  scoring  schemes  for  Ques¬ 
tions  1  and  2  are  presented  in  detail  in  Tables  2  and  3,  respectively. 
As  mentioned  above,  for  Question  1,  participants  were  given  a  sce¬ 
nario,  map,  and  table  of  data,  and  were  asked  to  identify  trends  in 
either  the  POL  distribution  or  personnel  replacement  system.  They 
were  scored  on  their  success  in  finding  five  "problematic  trends”  in 
the  data  table.  One  point  was  assigned  for  each  trend  correctly  iden¬ 
tified,  yielding  a  total  possible  score  of  5. 

The  scoring  of  the  COA  analysis  question  (Question  2)  was  more 
complex.  For  this  question,  participants  referred  to  the  original 
scenario,  the  original  map,  and  an  alternative  (direct- delivery) 
scenario.  They  were  asked  to  list  the  factors  they  would  consider  in 
evaluating  a  potential  switch  from  the  hub-and-spoke  system  to  a 
pure  direct- delivery  system. 

Before  delivering  the  training,  we  split  the  potential  set  of  responses 
to  Question  2  into  the  ten  categories  listed  in  Table  3.  Participants 
were  given  a  maximum  of  one  point  per  category,  regardless  of  the 
number  of  responses  they  provided  for  that  category.  So,  for  exam¬ 
ple,  five  responses  addressing  the  availability  of  resources  would 
yield  only  one  point. 

Three  raters  scored  the  tests.  First,  20  tests  were  scored  by  the  three 
raters  as  a  group  to  help  ensure  that  all  raters  agreed  on  the  applica¬ 
tion  of  the  scoring  scheme.  The  remaining  tests  were  split  into  three 
groups,  and  each  group  of  tests  was  assigned  to  two  of  the  three 
raters.  Thus,  each  rater  scored  two -thirds  of  the  tests  after  the  group¬ 
scoring  phase. 

After  the  scoring  was  completed,  a  factor  analysis  was  conducted  on 
the  ten  categories  of  responses  to  Question  2  to  determine  whether 
they  could  be  split  into  meaningful  subgroups.  As  a  result  of  the 
factor  analysis,  the  responses  were  grouped  into  three  separate  mea¬ 
sures:  (2a)  identifying  impacts  of  trends  in  data,  (2b)  coordinating 
and  influencing,  and  (2c)  tactics  and  resources.  The  responses  in  the 
final  category — tactics  and  resources — were  included  after  pilot  data 
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Table  2 

Question  1:  Identifying  Trends  in  Data 


Number  of 
Points 

Trend 

Example 

(from  POL  Vignette) 

1 

Bottleneck  at  the  hub 

Buildup  of  supply  at  Akan 

1 

Underutilized  node 

Kitami  is  consistently  underutilized 

1 

Starving  customers 

Supply  level  at  Niufu  DISCOM  is  low 

1 

Fluctuating  around 
maximum 

Port  Urohoro  around  maximum 

1 

Maximum:  5 

Growing  to  maximum 

Kamikawa  FSSP  reaching  capacity 

Table  3 

Question  2:  Analyzing  Courses  of  Action 

Number  of 

Response  Category: 

Points 

Response  Group 

Does  the  Policy . . . 

2a 

Identifying  impacts  of  trends 

1 

Alleviate  the  current  problem? 

1 

Have  impacts  on  customer? 

1 

Have  impacts  on  sources  of  supply 
or  intermediate  supply  points? 

1 

Maximum:  4 

Have  different  short-term  and 
long-term  impacts? 

2b 

Coordinating  and  influencing 

1 

Integrate  with  policies  of  other 
support  system  components 
(e.g.,  HNS,  bilateral  ally,  etc.)? 

1 

Exercise  influence  on  higher 
and/or  lower  echelons? 

1 

Agree  with  overall  mission? 

1 

Maximum:  4 

Introduce  tradeoffs  when 
compared  to  alternatives? 

2c 

Tactics  and  resources 

1 

Have  tactical  implications? 

1 

Have  impacts  on  resources  in 
theater? 

Maximum:  2 
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were  collected  in  an  effort  to  acknowledge  the  importance  of  tradi¬ 
tional  Army  concerns;  however,  the  category  was  not  an  explicit  part 
of  our  training  goals  for  this  demonstration  session. 

One  can  think  of  the  various  dependent  measures  on  a  continuum  of 
complexity.  Question  1  tests  the  simplest  skill:  identifying  trends  in 
data.  Question  2a,  identifying  impacts  of  trends,  builds  on  the  first 
skill  to  begin  quantitatively-based  analyses  of  courses  of  action.  That 
is,  a  logistician  must  first  be  able  to  make  sense  of  data  before  mov¬ 
ing  on  to  the  implications  of  those  data.  Finally,  once  the  impacts  of 
trends  are  determined,  the  logistician  can  consider  influencing  and 
coordinating  with  other  support  system  components  (Question  2b) 
to  achieve  a  desired  course  of  action. 

The  demonstration  session  and  the  microworld  model  used  in  the 
session  were  aimed  most  strongly  at  the  first  two  skills.  The  third 
skill — coordinating  and  influencing — was  mentioned,  but  was  not 
emphasized. 


Chapter  Three 

RESULTS 


The  learners'  performance  on  the  objective  questions  as  well  as  their 
subjective  ratings  of  their  own  understanding  suggest  that  partici¬ 
pants  acquired  a  significant  amount  of  knowledge  and  skill  in  distri¬ 
bution  management  as  a  result  of  participating  in  the  training 
session. 

OBJECTIVE  MEASURES 

Before  proceeding  with  analyses  of  participants’  performance  on  the 
objective  questions,  we  calculated  inter-rater  reliabilities.  The  mean 
correlation  between  pairs  of  raters’  scores  was  0.80,  leaving  us  rea¬ 
sonably  confident  that  the  three  raters  applied  the  scoring  scheme 
equivalently.  In  all  further  analyses  of  objective  test  data,  we  used 
the  average  of  the  scores  assigned  by  the  two  raters  for  each  test. 

Only  two  participants  failed  to  answer  Question  3  (our  manipulation 
check  testing  participants’  ability  to  read  the  data  tables)  correctly  on 
the  pretest,  and  both  participants  answered  a  similar  question  cor¬ 
rectly  on  the  posttest.  This  suggests  that  they  acquired  this  basic  skill 
during  the  training  session,  and  their  data  were  therefore  combined 
with  the  rest  of  the  sample. 

Participants  showed  significant  learning  of  the  two  skills  emphasized 
in  the  demonstration  session  (see  Figure  10).  On  average,  learners 
were  able  to  identify  one  more  trend  out  of  a  possible  total  of  five 
after  participating  in  the  session  (Question  1:  A  =  0.91,  f(51)  ~  5.93, 
p  <  .0005).  Gains  on  the  second  skill— identifying  impacts  of  trends— 
were  smaller,  but  still  reliable  (Question  2a:  A  =  0.47,  t{ 51)  =  4.00, 
p  <  .0005). 
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Figure  10 — Performance  of  the  Full  Assessment  Group  on  Question  1 
(Identifying  Trends)  and  Question  2a  (Identifying  Impacts 
of  Courses  of  Action) 


Skills  that  were  not  the  explicit  targets  of  the  training  curriculum  (i.e., 
coordinating  and  influencing,  and  tactics  and  resources)  did  not 
improve  significantly  (Question  2b:  A  =  0.10,  f(51)  =  0.90,  p  >  .05; 
Question  2c:  A  =  -0.04,  f(51)  =  -0.68,  p  >  .05).  The  mean  posttest 
scores  for  all  four  measures  fell  in  the  low  to  middle  part  of  the  range 
for  each  measure  and  were  as  follows:  M(Q1  posttest)  =  1.84,  M(Q2a 
posttest)  =  1.17,  M(Q2b  posttest)  =  0.73,  and  M(Q2c  posttest)  =  0.75. 

Based  on  the  results  of  the  f-tests,  we  restricted  all  further  analysis  to 
Questions  1  and  2a.  Both  in  identifying  trends  and  identifying 
impacts  of  trends,  participants  showed  larger  gains  for  the  POL 
vignette  than  for  the  personnel  vignette;  however,  the  differences 
between  vignettes  failed  to  reach  significance.  For  Question  1,  the 
mean  improvements  on  the  POL  and  personnel  vignettes  were  1.13 
and  0.69,  respectively  (f( 50)  =  1.45,  p  >  .05).  The  mean  posttest- 
pretest  difference  scores  for  Question  2a  were  0.58  for  POL  and  0.36 
for  personnel  (f(50)  =  0.93,  p  >  .05).  It  is  worth  noting  that  the  POL 
vignette  is  arguably  a  closer  analog  to  the  general  supply  vignette 
used  during  the  training  session  than  is  the  personnel  vignette.  Thus 
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there  may  have  been  more  effective  transfer  of  learning  to  the  POL 
vignette,  explaining  the  consistent,  albeit  nonsignificant,  difference 
between  the  means  for  the  two  test  vignettes. 

After  conducting  the  preliminary  t- tests  reported  above,  we  pro¬ 
duced  a  correlation  matrix  to  identify  the  variables  that  were  related 
to  our  two  main  outcome  measures.  The  following  variables  were 
included  in  the  correlation  matrix:  (1)  outcome  measures — pretest 
and  posttest  scores  for  each  of  the  objective  questions,  posttest- 
pretest  difference  scores;  (2)  experimental  control  variables — 
vignette  (POL  or  personnel),  test  form  (A  or  B),  and  test  order  (AB  or 
BA);  and  (3)  demographic  variables — rank,  time  served  in  permanent 
unit,  time  served  in  current  position,  degree  of  civilian  education, 
enlisted  versus  officer,  and  prior  training  (i.e.,  participation  in  related 
pilot  studies). 

None  of  the  variables  correlated  even  moderately  with  the  posttest- 
pretest  difference  scores;  however,  a  few  demographic  variables 
looked  as  though  they  varied  with  either  pretest  performance, 
posttest  performance,  or  both.  We  included  those  variables — civilian 
education,  enlisted/ officer,  and  prior  training — in  a  regression 
model.  The  results  of  the  regressions  are  presented  in  Tables  4  and  5. 

The  level  of  civilian  education  had  no  significant  effect  on  either 
pretest  or  posttest  performance  for  either  measure.  However,  as 
shown  in  Figure  11,  both  on  the  pretest  and  on  the  posttest,  officers 
were  better  at  identifying  trends  in  data  than  were  enlisted  partici¬ 
pants  [M  (officer,  pretest)  =  1.18,  M(enlisted,  pretest)  =  0.30, 
M(officer,  posttest)  =  2.16,  M(enlisted,  posttest)  =  1.03].  The  two 
groups  showed  statistically  equivalent  gains  from  pretest  to  posttest. 
A  similar  pattern  was  seen  for  Question  2a  [M(officer,  pretest)  =  0.93, 
M(enlisted,  pretest)  =  0.16,  M(officer,  posttest)  =  1.37,  M(enlisted, 
posttest)  =  0.69];  however,  the  effect  was  not  statistically  significant 
for  the  posttest. 

Eight  learners  had  previously  participated  in  RAND  pilot  studies  on 
topics  similar  to  those  presented  during  the  demonstration  training 
session.  Although  there  was  evidence  that  participants  with  prior 
training  experience  knew  more  at  the  start  of  the  session  and  also  at 
the  end  of  the  session,  they  did  not  make  bigger  gains  in  perfor¬ 
mance  than  did  other  participants.  On  Question  1,  participants  with 
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Table  4 


Summary  of  Regression  Analysis  for  Variables 
Predicting  Pretest  Performance 
(AT  =52) 


Ql:  Identifying  Trends 

Q2a:  Identifying  Impacts 

Variable 

B 

SE(B) 

B 

SE(B) 

Test  vignette 

0.02 

0.23 

-0.18 

0.19 

Civilian  education 

0.04 

0.13 

-0.02 

0.10 

Enlisted /officer 

-0.66 

0.30* 

-0.65 

0.24* 

Prior  RAND  training 

0.64 

0.31* 

0.59 

0.26* 

*p  <  .05. 

Table  5 

Summary  of  Regression  Analysis  for  Variables 

Predicting  Posttest  Performance 

(AT  =52) 

Ql:  Identifying  Trends 

Q2a:  Identifying  Impacts 

Variable 

B 

SE(B) 

B 

SE(B) 

Test  vignette 

0.46 

0.36 

0.05 

0.26 

Civilian  education 

0.13 

0.20 

0.09 

0.14 

Enlisted/officer 

-0.82 

0.47* 

-0.27 

0.34 

Prior  RAND  training 

0.65 

0.50 

1.09 

0.35* 

*p  <  .05. 


prior  training  scored  higher  on  both  the  pretest  and  the  posttest  than 
other  participants  did  [M(prior  training,  pretest)  =  1.69,  M(no  prior 
training,  pretest)  =  0.78,  M(prior  training,  posttest)  =  2.63,  M(no  prior 
training,  posttest)  =  1.69],  but  the  difference  between  the  two  groups 
was  significant  only  for  the  pretest.  As  shown  in  Figure  12,  partici¬ 
pants  with  prior  RAND  training  experience  were  significantly  better 
at  identifying  impacts  of  courses  of  action,  both  on  the  pretest  and 
on  the  posttest,  than  were  those  without  prior  training  [M(prior 
training,  pretest)  =  1.42,  M(no  prior  training,  pretest)  =  0.58,  M(prior 
training,  posttest)  =  2.25,  M(no  prior  training,  posttest)  =  0.98]. 
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Figure  1 1— -Performance  of  Officers  and  Enlisted  Soldiers  on  Question  1 
(Identifying  Trends) 
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Figure  12— Performance  of  Participants  With  and  Without  Prior  RAND 
Training  on  Question  2a  (Identifying  Impacts  of  Courses  of  Action) 
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Posttest  data  from  the  Posttest  Control  group  were  compared  with 
data  from  the  Full  Assessment  group  to  determine  whether  experi¬ 
ence  with  the  pretest  itself  had  any  effect  on  subsequent  posttest 
performance  (see  Figure  13).  There  was  no  evidence  of  an  effect  of 
pretest  experience  on  the  posttest  (f(50)  =  0.46,  p  >  .05  for  Question  1; 
f(50)  =  0.49,  p  >  .05  for  Question  2a). 

SUBJECTIVE  MEASURES 

Consistent  with  the  objective  measures,  participants  believed  their 
understanding  of  key  concepts  related  to  distribution  management 
increased  during  the  course  of  the  three-hour  training  session.  As 
shown  in  Figure  14,  on  a  scale  of  0  to  10,  they  rated  their  understand¬ 
ing  of  key  concepts  between  two  or  three  points  higher  after  training 
than  before  training.  Specifically,  participants  reported  increases  in 
their  understanding  of  how  to  measure  a  theater  distribution  system 
(A  =  3.00,  t{ 51)  =  12.29,  p  <  .0005),  how  to  evaluate  alternative  distri¬ 
bution  policies  (A  =  2.88,  t(51)  =  11.46,  p  <  .0005),  the  impacts  of 
policies  over  time  (A  =  2.21,  f(51)  =  7.51,  p  <  .0005),  the  impacts  of 
policies  on  other  organizations  (A  =  1.97,  f (51)  =  8.18,  p  <  .0005),  and 
how  to  manage  a  general  supply  distribution  network  (A  =  2.15,  t{ 51) 
=  7.73,  p<. 0005). 
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RAND  MR1305-13 


Score 

Figure  13— Posttest  Performance  of  the  Full  Assessment  Group  and 
Posttest  Control  Group  on  Questions  1, 2a,  2b,  and  2c 
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Figure  14— Changes  in  Participants’  Self-Ratings  of  Understanding  of  Key 
Concepts  Related  to  Distribution  Management 
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DISCUSSION 


ALL  GROUPS  LEARN  MANAGEMENT  SKILLS  UNDER 
DIFFICULT  ASSESSMENT  CRITERIA 

Foremost  among  this  study’s  findings  are  the  significant  improve¬ 
ments  in  logistics  managers’  ability  to  identify  trends  in  data  and 
interpret  the  effects  of  those  trends.  It  should  again  be  emphasized 
that  this  study  was  constructed  to  provide  a  challenging  environ¬ 
ment  for  learners.  The  authors  did  not  want  to  simply  "teach  to  the 
test.”  Learners  encountered  different  scenarios  for  learning  and  as¬ 
sessment.  How  the  data  were  formatted  also  differed  from  instruc¬ 
tion  (graphical)  to  assessment  (table  of  numbers).  Earlier  research 
on  learning  and  problem  solving  in  analogical  settings  (see,  e.g.,  Gick 
and  Holyoak,  1983)  has  shown  that  such  changes  to  the  contexts  in 
which  problems  are  presented  makes  transfer  from  one  problem 
(e.g.,  the  training  scenario)  to  other  problems  (e.g.,  the  assessment 
scenarios)  particularly  challenging. 

Also  of  note  is  the  amount  of  knowledge  we  attempted  to  teach  in  a 
very  short  time.  We  sought  to  teach  these  learners  a  new  way  of 
viewing  their  organizations  and  work.  Whereas  the  emphasis  in  the 
past  has  largely  been  on  monitoring  and  accounting  for  past  events, 
we  emphasized  systemic,  dynamic,  and  proactive  management.  At 
the  same  time,  we  sought  to  teach  very  specific  skills — finding  trends 
in  data  and  identifying  the  impacts  of  those  trends.  The  majority  of 
the  discussion  and  microworld  use  directed  specifically  toward  our 
teaching  goals  amounted  to  a  total  of  only  60  minutes  of  actual  time 
on  task.  The  other  two  hours  of  the  session  were  devoted  to  intro¬ 
ductions,  assessments,  a  short  break,  and  a  later  demonstration  of  a 
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feedback  loop  in  the  microworld  shown  after  the  assessment.  This  is 
very  little  time  in  which  to  cover  the  amount  of  material  we  sought  to 
teach.  We  were  asking  our  students  to  “drink  from  a  fire  hose”  of 
training  content,  and  they  clearly  learned  some  skills. 

Finally,  we  were  quite  interested  in  the  fact  that  all  groups  of  learners 
improved  their  skills.  Performance  increased  by  about  the  same 
amount  for  both  officers  and  enlisted  soldiers,  even  though  officers 
can  generally  be  presumed  to  have  higher  levels  of  formal  education. 
Trainees  with  previous  training  experience  in  these  topics  also  in¬ 
creased  their  skill  levels,  so  there  is  some  evidence  that  this  learning 
is  cumulative.  Overall,  the  small  but  significant  learning  results  jus¬ 
tify  some  optimism  regarding  such  training  methods  and  content. 
Moreover,  we  believe  actual  training  would  be  carried  out  in  ways 
that  would  yield  much  stronger  learning  results;  this  is  further  justifi¬ 
cation  for  our  optimism. 

IMPLEMENTING  MICROWORLD-BASED  TRAINING  UNDER 
OPTIMAL  LEARNING  CONDITIONS 

It  is  important  to  remember  that  our  study  was  implemented  to  meet 
the  constraints  of  quasi- experimental  conditions.  We  wanted  all  our 
trainees  to  see  the  same  materials  and  be  exposed  to  the  same  explo¬ 
rations.  These  constraints  run  directly  counter  to  some  of  the  basic 
strengths  of  exploratory  learning  via  microworlds.  Under  more  ideal 
conditions,  learners  interact  with  the  microworlds  either  individually 
or  in  small  groups.  Hypotheses  are  generated,  predictions  are  made 
about  system  performance  under  certain  conditions  (both  short-  and 
long-term  outcomes  as  well  as  side  effects),  and  the  simulation  is 
run.  The  results  of  the  simulation  run  are  compared  to  the  predic¬ 
tions  and  then  conclusions  are  drawn,  new  hypotheses  are  gener¬ 
ated,  and  new  explorations  are  planned.  This  exploratory  cycle  can 
be  fast  or  slow,  depending  on  the  results  of  the  explorations  and  the 
ensuing  discussions. 

The  conditions  of  this  study  included  an  introduction  to  the  micro¬ 
worlds,  including  the  most  basic  explanations  of  the  graphics  and 
animation.  For  example,  trainees  learned  what  the  transportation 
assets  looked  like  when  they  were  full  versus  empty,  what  the  bar 
graphs  denoted,  and  how  to  parse  the  overall  flows  of  resources  as 


Discussion  33 


the  simulation  was  run.  The  costs  of  time  spent  learning  these  basics 
of  how  to  understand  the  microworld  would  be  amortized  over  more 
sessions  in  actual  training  use. 


ASSESSMENTS  ADDRESS  A  SUBSET  OF  TRAINING 
CONTENT 

We  sought  to  assess  several  aspects  of  how  a  complex  distribution 
system  is  understood  and  managed.  This  includes  the  larger  con¬ 
cepts  of  how  to  view  systems  as  interrelated,  interacting  sets  of  enti¬ 
ties  that  jointly  produce  a  business  process.  It  also  includes  a 
“customer”  orientation,  how  to  think  about  end-to-end  system  per¬ 
formance,  and  how  to  design  appropriate  metrics  to  assess  how  the 
system  is  performing.  As  mentioned  earlier,  these  are  new  concepts 
to  most  of  the  trainees  in  the  context  of  Army  logistics  operations. 

In  conducting  the  study,  we  faced  serious  constraints  on  the  amount 
of  assessment  time.  We  believe  that  trainees  learned  more  than  we 
had  the  time  to  assess.  In  debriefings  and  discussions  with  trainees 
after  our  pilot  sessions  and  the  study  reported  here,  it  appears  they 
acquired  a  deeper  understanding  of  how  to  think  more  systemically 
and  be  more  proactive  in  their  planning.  This  differs  significantly 
from  the  traditional  “reactive”  management  strategies  under  which 
they  normally  train.  Comments  about  trainees  learning  to  lengthen 
their  planning  horizons  were  not  uncommon.  This  result  is  backed 
by  the  trainees'  self-reports  of  their  knowledge  before  and  after  the 
study  as  reported  in  Chapter  Three.  Their  ratings  of  their  under¬ 
standing  of  important  topics  increased  by  a  minimum  of  40  percent 
and,  in  one  case,  nearly  100  percent  from  pretest  to  posttest. 

FOLLOWING  UP  WITH  DIFFERENT  CONTROL  CONDITIONS 

The  simple  control  condition  for  this  study  established  that  there  was 
no  significant  learning  from  taking  the  pretest:  Trainees  who  did  not 
take  the  pretest  performed  no  differently  than  trainees  who  did. 
However,  there  are  a  number  of  other  appropriate  control  conditions 
that  could  answer  questions  about  how  microworld-based  training 
compares  to  other  training  methods. 
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One  of  the  problems  in  assessing  new  uses  of  technology  to  support 
learning  is  that  new  training  methods  like  microworlds  are  not  im¬ 
plemented  in  a  vacuum.  As  in  this  study,  the  new  methods  are  used 
because  we  want  to  teach  new  content  effectively.  Other  methods 
(e.g.,  discussions)  and  other  pieces  of  the  curriculum  are  added  to 
support  the  microworld  use.  It  therefore  becomes  difficult  to  isolate 
the  effects  of  the  microworld  exposure. 

Complicating  matters  in  the  context  of  this  study  is  the  fact  that  there 
are  no  other  current  curricula  to  teach  the  content  for  comparison. 
These  are  new  concepts  to  the  Army  logistics  community,  so  there  is 
no  natural  set  of  materials  and  methods  to  use  as  a  control  condition. 
This  is  unfortunate,  because  a  demonstration  of  the  superiority  of  a 
curriculum  using  a  microworld  model  over  the  equivalent  curricu¬ 
lum  without  such  a  model  would  clearly  provide  the  strongest  sup¬ 
port  for  the  use  of  microworlds  in  the  training  of  logistics  skills. 

Given  the  availability  of  alternative  curricula,  potentially  useful  con¬ 
trol  conditions  could  teach  the  same  content  but  not  use  dynamic 
simulations.  They  might  instead  involve  the  use  of  pictures  of  the 
distribution  networks  coupled  with  discussions  of  the  different  hub- 
and-spoke  versus  direct- delivery  scenarios.  Another  option  would 
be  for  the  Army  to  design  curricula  to  teach  the  same  content  and 
compare  the  performance  of  the  two  groups  on  the  same  pretests 
and  posttests.  Care  must  be  taken  in  such  comparisons  that  the  cur¬ 
ricula  and  the  instructors  are  not  simply  teaching  to  the  test. 


FUTURE  DIRECTIONS:  DISTRIBUTED,  COLLABORATIVE 
MICROWORLD  WITH  REUSABLE  LIBRARIES  OF  SCENARIOS 

If  microworld-based  curricula  continue  to  demonstrate  promise  as 
effective  methods  for  teaching  complex  logistics  management  skills, 
a  number  of  directions  could  be  pursued  to  support  their  use.  First, 
reusable  libraries  of  scenarios  could  be  developed  and  accessed  via 
networks.  In  fact,  this  project  is  already  developing  microworlds  to 
help  teach  the  skills  that  we  did  not  cover  in  depth,  namely,  how  to 
coordinate  with  and  influence  other  organizations.  We  developed  a 
microworld  that  simulates  the  process  of  designing  and  building  the 
distribution  network  we  used  as  the  focus  of  our  study.  Key  to  this 
microworld  are  learning  the  information  and  command  flows 
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between  organizations  and  how  the  actions  of  different  organiza¬ 
tions  or  environmental  events  can  have  effects  on  the  design  and 
construction  process.  This  microworld  is  linked  systemically  to  the 
activities  of  the  distribution  network  as  it  evolves:  If  the  pieces  of  the 
deployed  distribution  network  are  not  completed  on  time,  materiel 
starts  to  back  up  at  the  ports  and  becomes  vulnerable. 

Having  networked  access  to  microworld-based  curricula  (via  the 
World  Wide  Web,  in  the  most  general  form)  would  also  allow  access 
to  other  collaborative  problem-solvers  from  both  shared  and  sister 
organizations.  Tools  to  support  such  distributed  use  of  microworlds 
are  already  being  made  available  by  the  developers  of  tools  to  build 
microworlds.  This  would  allow  trainees  in  geographically  distributed 
locations  to  work  together  simultaneously. 

We  believe  that  there  is  a  great  deal  of  promise  for  the  effectiveness 
of  microworld-based  curricula  for  training  complex  management 
skills.  The  comments  and  ratings  of  the  trainees  who  participated  in 
our  microworld-based  training  sessions  support  our  optimism.  Over 
the  course  of  their  careers,  Army  personnel  have  been  exposed  to  a 
great  deal  of  training,  both  good  and  bad.  Hence,  they  are  critical 
consumers  of  new  training  and  tend  to  be  very  direct  about  their 
opinions.  Their  views  of  the  current  and  potential  usefulness  of 
microworld-based  training  methods  and  curricula,  as  reported  in 
their  free  responses,  were  very  positive.  Part  of  the  post-training 
assessment  asked  trainees  to  rate  the  effectiveness  of  the  simulations 
in  “developing  your  understanding"  on  a  number  of  topics  we  sought 
to  teach.  The  ratings  were  either  “not  at  all  useful,"  “somewhat  use¬ 
ful,”  “useful,"  or  “very  useful."  Thirty-nine  percent  of  the  trainees 
rated  the  simulations  as  “useful"  and  44  percent  rated  them  as  “very 
useful." 

Microworld-based  curricula  appear  to  offer  reasonable  solutions  to 
the  challenges  faced  by  the  Army  in  training  logistics  managers. 
They  offer  dynamic,  semirealistic  simulated  distribution  systems  to 
support  exploratory  learning.  They  can  focus  on  teaching  specific, 
job-critical  skills  without  waiting  for  larger-scale  training  exercises, 
with  either  locally  gathered  or  distributed  groups  of  learners.  They 
appear  to  require  fewer  resources  to  build  and  use  than  other  cur¬ 
rent,  large  simulation-based  learning  tools.  If  these  promises 
continue  to  hold  true,  microworld-based  curricula  will  be  strong 
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candidates  for  training  a  wide  variety  of  management  skills  in  both 
military  and  commercial  settings. 


REFERENCES 


Bondanella,  J.  R„  Lewis,  M.  W.,  Steinberg,  P.  S.,  Park,  G.  S.,  Sollinger, 
J.  M.,  Winkler,  J.  D.,  Halliday,  J.  H.,  and  Way-Smith,  S.  (1999). 
Microworld  Simulations  for  Command  and  Control  Training  of 
Theater  Logistics  and  Support  Staffs:  A  Curriculum  Strategy.  Santa 
Monica,  CA:  RAND,  MR-929 -A. 

Bondanella,  J.  R.,  Lewis,  M.  W.,  Park,  G.  S.,  Winkler,  J.  D.,  and 
Sollinger,  J.  M.  "Support  Unit  C2  Training  for  Force  XXI:  A 
Process  Approach.”  Unpublished  RAND  research,  1997. 

Cohen,  D.  K.  (1988).  "Educational  Technology  and  School  Organiza¬ 
tion.”  In  R.  S.  Nickerson  and  P.P  .  Zohaites  (eds.),  Technology  in 
Education:  Looking  Toward  2020.  Hillsdale:  Erlbaum,  pp.  231- 
264. 

Cuban,  L.  (1986).  Teachers  and  Machines.  New  York:  Teachers  Col¬ 
lege  Press. 

Gick,  M.  L.,  and  Holyoak,  K.  J.  (1983).  “Schema  Induction  and 
Analogical  Transfer.  Cognitive  Psychology,  Vol.  15,  pp.  1-38. 

Gross,  D.  (1996).  “Endless  Possibilities— Tools  That  Simulate  'What 
If  Scenarios  Show  Managers  That  a  Picture  Is  Worth  a  Thousand 
Words.”  CIO  Magazine,  February. 

Hall,  B.  (1995).  Return- on- Investment  and  Multimedia  Training.  San 
Francisco,  CA:  Macromedia,  Inc. 

Lawler,  R.  W.  (1987).  “Learning  Environments:  Now,  Then  and 
Someday.”  In  R.  W.  Lawler  and  M.  Yazdani  (eds.),  Artificial Intelli- 


37 


38  Exploring  Microworld  Models  to  Train  Army  Logistics  Management  Skills 


gence  and  Education,  Volume  1:  Learning  Environments  and 
Tutoring  Systems.  Norwood,  NJ:  Ablex. 

Mandinach,  E.  B.  (1989).  "Model-Building  and  the  Use  of  Computer 
Simulations  of  Dynamic  Systems.”  Journal  of  Educational  Com¬ 
puting  Research,  Vol.  5,  No.  2,  pp.  221-243. 

Mandinach,  E.  B.,  and  Cline,  H.  F.  (1994).  Classroom  Dynamics:  Im¬ 
plementing  a  Technology-Based  Learning  Environment.  Hillsdale: 
Erlbaum. 

McArthur,  D.,  and  Lewis,  M.  (1991).  Overview  of  Object  Oriented 
Microworlds  for  Learning  Mathematics  Through  Inquiry.  Santa 
Monica,  CA:  HAND,  N-3242-NSF. 

McArthur,  D.,  and  Lewis,  M.  W.  (1998).  Muddy  Learning:  Evaluating 
Learning  in  Multi-User  Computer-Based  Environments.  Santa 
Monica,  CA:  RAND,  DRU-1688-DARPA,  February. 

McArthur,  D.,  Lewis,  M.,  and  Bishay,  M.  (1995).  “ESSCOTS  for 
Learning:  Transforming  Commercial  Software  into  Powerful  Edu¬ 
cational  Tools.  Journal  of  Artificial  Intelligence  and  Education, 
Vol.  6,  No.  1,  pp.  3-33. 

Papert,  S.  (1980).  Mindstorms:  Children,  Computers,  and  Powerful 
Ideas.  New  York:  Basic  Books. 

Richmond,  B.  (1993).  “Systems  Thinking:  Critical  Thinking  Skills  for 
the  1990s  and  Beyond.”  Systems  Dynamics  Review,  Vol.  9,  No.  2, 
pp.  113-133. 

Salopek,  J.  J.  (1998).  Workstation  Meets  Playstation,  Training  and 
Development  Magazine,  August. 

Senge,  P.  M.,  Roberts,  C.,  Ross,  R.  B.,  Smith,  B.  J.,  and  Kleiner,  A. 
(1994).  The  Fifth  Discipline  Fieldbook :  Strategies  and  Tools  for 
Building  a  Learning  Organization.  New  York:  Doubleday. 

Shute,  V.,  and  Glaser,  R.  (1990).  “Large-Scale  Evaluation  of  an  Intelli¬ 
gent  Discovery  World:  SMITHTOWN.”  Interactive  Learning  Envi¬ 
ronments,  Vol.  1,  pp.  51-77. 


References  39 


Shute,  V.,  Glaser,  R.,  and  Raghavan,  K.  (1988).  Inference  and  Discov¬ 
ery  in  an  Experimental  Laboratory .  Pittsburgh:  LRDC  Technical 
Report  No.  10. 

Smith,  R.  (1986).  “The  Alternate  Reality  Kit:  An  Animated  Environ¬ 
ment  for  Creating  Simulations.”  Proceedings  of  the  1986  IEEE 
Computer  Society  Workshop  on  Visual  Languages ,  pp.  99-106. 

Starr,  P.  (1994).  “Seductions  of  Sim:  Policy  as  a  Simulation  Game.” 
The  American  Prospect ,  No.  17,  Spring,  pp.  19-29. 

White,  B.  Y.,  and  Horowitz,  P.  (1987).  Thinkertools:  Enabling  Chil¬ 
dren  to  Understand  Physical  Laws.  Cambridge,  MA:  BBN  Labora¬ 
tories,  Report  No.  6470. 


Exploring  the  Use  of  Microworld  Models 
to  Train  Army  Logistics  Management 


The  authors  performed  a  study  to  assess  the  effectiveness  of  using  microworlds 


to  train  Army  logisticians.  Sponsored  by  the  U.S.  Army  Combined  Arms 
Support  Command,  the  study  was  part  of  an  effort  to  design  new  training  for  an 
emerging  organization  in  the  Army — the  Theater  Support  Command  (TSC) — 
which,  among  other  things,  is  responsible  for  managing  the  flow  of  people  and 
materiel  throughout  the  theater  of  operations.  The  authors  designed  a  three- 
hour  training  curriculum  around  a  microworld  model  that  represents  a  simpli¬ 
fied,  but  dynamic,  model  of  the  distribution  management  process.  The  main 
goal  of  the  session  was  to  teach  the  learners  about  the  consequences  of  their 
decisions  over  time  and  across  the  distribution  management  system.  Participants 
in  the  microworld-based  training  sessions  improved  significantly  in  their  ability 
to  identify  problematic  trends  in  a  distribution  network  and  to  evaluate  the 
impacts  of  those  trends. 
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